{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# 自定义数据\n",
    "\n",
    "[![](https://gitee.com/mindspore/docs/raw/master/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/master/tutorials/source_zh_cn/middleclass/custom/data.ipynb)\n",
    "\n",
    "MindSpore可以加载常见的数据集或自定义的数据集，这部分功能在初级教程中进行了部分介绍。加载自定义数据集有两种途径：\n",
    "\n",
    "- 通过`GeneratorDataset`对象加载，使用方法可参考[初级教程-自定义数据集](https://www.mindspore.cn/tutorial/zh-CN/master/dataset.html#%E8%87%AA%E5%AE%9A%E4%B9%89%E6%95%B0%E6%8D%AE%E9%9B%86)。\n",
    "\n",
    "- 将数据集转换为MindRecord，即MindSpore数据格式，通过读取MindRecord文件进行加载数据。\n",
    "\n",
    "如果用户想要获得更好的性能体验，可以将数据集转换为MindRecord，从而方便地加载到MindSpore中进行训练。\n",
    "\n",
    "MindRecord的性能优化如下：\n",
    "\n",
    "- 实现多变的用户数据统一存储、访问，训练数据读取更加简便。\n",
    "- 数据聚合存储，高效读取，且方便管理、移动。\n",
    "- 高效的数据编解码操作，对用户透明、无感知。\n",
    "- 可以灵活控制分区的大小，实现分布式训练。\n",
    "\n",
    "常见数据集转换MindRecord可参考官方编程指南中的[MindSpore数据格式转换](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/dataset_conversion.html#MindSpore%E6%95%B0%E6%8D%AE%E6%A0%BC%E5%BC%8F%E8%BD%AC%E6%8D%A2)，自定义数据集转换可参考下文。\n",
    "\n",
    "MindRecord的目标是归一化用户的数据集，并进一步通过`MindDataset`实现数据的读取，用于训练过程。下面对这两步进行说明。\n",
    "\n",
    "## 自定义数据集转换为MindRecord\n",
    "\n",
    "首先，下载需要处理的图片数据`transform.jpg`作为待处理的原始数据。\n",
    "\n",
    "创建文件夹目录`./datasets/convert_dataset_to_mindrecord/datas_to_mindrecord/`用于存放所有的转换数据集。\n",
    "\n",
    "创建文件夹目录`./datasets/convert_dataset_to_mindrecord/images/`用于存放下载下来的图片数据。\n",
    "\n",
    "在Jupyter Notebook中执行以下命令，完成图片下载和文件夹的创建，并将图片移动到指定位置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./datasets/convert_dataset_to_mindrecord/images/\n",
      "└── transform.jpg\n",
      "\n",
      "0 directories, 1 file\n"
     ]
    }
   ],
   "source": [
    "!wget -N --no-check-certificate https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/transform.jpg\n",
    "!mkdir -p ./datasets/convert_dataset_to_mindrecord/datas_to_mindrecord/\n",
    "!mkdir -p ./datasets/convert_dataset_to_mindrecord/images/\n",
    "!mv -f ./transform.jpg ./datasets/convert_dataset_to_mindrecord/images/\n",
    "!tree ./datasets/convert_dataset_to_mindrecord/images/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "导入文件写入工具类`FileWriter`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from mindspore.mindrecord import FileWriter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建FileWriter对象，传入文件名及分片数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "data_path = './datasets/convert_dataset_to_mindrecord/datas_to_mindrecord/'\n",
    "data_record_path = './datasets/convert_dataset_to_mindrecord/datas_to_mindrecord/test.mindrecord'\n",
    "writer = FileWriter(file_name=data_record_path,shard_num=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "定义数据集结构文件Schema，调用`write_raw_data`接口写入数据，最后调用`commit`接口生成本地数据文件。\n",
    "\n",
    "Schema文件主要包含字段名`name`、字段数据类型`type`和字段各维度维数`shape`：\n",
    "\n",
    "- 字段名：字段的引用名称，可以包含字母、数字和下划线。\n",
    "\n",
    "- 字段数据类型：包含int32、int64、float32、float64、string、bytes。\n",
    "\n",
    "- 字段维数：一维数组用[-1]表示，更高维度可表示为[m, n, …]，其中m、n为各维度维数。\n",
    "\n",
    "> 如果字段有属性`shape`，则用户传入`write_raw_data`接口的数据必须为`numpy.ndarray`类型，对应数据类型必须为int32、int64、float32、float64。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MSRStatus.SUCCESS"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 定义schema\n",
    "data_schema = {\"file_name\":{\"type\":\"string\"},\"label\":{\"type\":\"int32\"},\"data\":{\"type\":\"bytes\"}}\n",
    "writer.add_schema(data_schema,\"test_schema\")\n",
    "\n",
    "# 数据准备\n",
    "file_name = \"./datasets/convert_dataset_to_mindrecord/images/transform.jpg\"\n",
    "with open(file_name, \"rb\") as f:\n",
    "    bytes_data = f.read()\n",
    "data = [{\"file_name\":\"transform.jpg\", \"label\":1, \"data\":bytes_data}]\n",
    "\n",
    "indexes = [\"file_name\",\"label\"]\n",
    "writer.add_index(indexes)\n",
    "\n",
    "# 数据写入\n",
    "writer.write_raw_data(data)\n",
    "\n",
    "# 生成本地数据\n",
    "writer.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该示例会生成8个文件，成为MindRecord数据集。`test.mindrecord0`和`test.mindrecord0.db`称为1个MindRecord文件，其中`test.mindrecord0`为数据文件，`test.mindrecord0.db`为索引文件，生成的文件为：\n",
    "\n",
    "```\n",
    "\n",
    "./datasets/convert_dataset_to_mindrecord/datas_to_mindrecord/\n",
    "├── test.mindrecord0\n",
    "├── test.mindrecord0.db\n",
    "├── test.mindrecord1\n",
    "├── test.mindrecord1.db\n",
    "├── test.mindrecord2\n",
    "├── test.mindrecord2.db\n",
    "├── test.mindrecord3\n",
    "└── test.mindrecord3.db\n",
    "\n",
    "0 directories, 8 files\n",
    "```\n",
    "\n",
    "## 读取MindRecord数据集\n",
    "\n",
    "导入读取类`MindDataset`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import mindspore.dataset as ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先使用`MindDataset`读取MindRecord数据集，然后对数据创建字典迭代器，并通过迭代器读取一条数据记录。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample: {'data': array([255, 216, 255, ..., 159, 255, 217], dtype=uint8), 'file_name': array(b'transform.jpg', dtype='|S13'), 'label': array(1, dtype=int32)}\n",
      "Got 1 samples\n"
     ]
    }
   ],
   "source": [
    "file_name = './datasets/convert_dataset_to_mindrecord/datas_to_mindrecord/test.mindrecord0'\n",
    "# create MindDataset for reading data\n",
    "define_data_set = ds.MindDataset(dataset_file=file_name)\n",
    "# create a dictionary iterator and read a data record through the iterator\n",
    "count = 0\n",
    "for item in define_data_set.create_dict_iterator(output_numpy=True):\n",
    "    print(\"sample: {}\".format(item))\n",
    "    count += 1\n",
    "print(\"Got {} samples\".format(count))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore-1.3.0",
   "language": "python",
   "name": "mindspore-1.3.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
